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Agenda 


Human performance envelope? 


Past research: 
— Research motivation & overview 
— Initial findings 


Present activities: Confirmation and extension 
— What happens when controllers are working with automation? Overview 


Future directions 
Conclusions 


Applications 


Human performance envelope 


At edges, due to difficulty, 
complexity, overload etc. 
performance/safety may 
be temporarily 
compromised; but 
situation normally 
recovered before loss of 
separation event 


Normal operations: 
ATC is working 
effectively within this 
workload and 
scenario space 


Situation 
Awareness 


Here a loss 
of separation 
will occur 


Fatigue Workload 


Stress Memory 


/ Attention 


Communications 


Teamwork malas 


Motivation 


ATM is an ‘ultra-safe’ industry 


ATM remains highly “‘human-centric’ — real-time 
operations 


Mitigations defend against incidents, but still 
occur 


Need to know when controllers are 
approaching the edges of acceptable 
performance 


Research overview 


¢ Overall Aims 
— Identify factors 
— Identify and verify interactions that threaten performance 
— Develop markers of performance limits or boundaries 


e Potential Outcomes 


— Better understanding of ‘difficult’ human performance factors in 
Air Traffic Control (ATC) 


— Signs and symptoms that performance is threatened 


Study approach 
2009 
reports 


Interacting 
Sim. trials factors 
\Y 
PAU Controller Markers of 
— Tht interview performance 


Factor 
identification 


limits 


Method: ATC exercise 


¢ Design 


IV: Taskload Covariates: Arousal, Fatigue DV: (Performance): STCA, route 
(low/high) SA, Stress, Workload directs, time to respond 


° Measures 


Measure Stress—Arousal Visual a Present SACL an 
Check List Analogue Assessment Method Self Assessment 


Interval 
(Mins) 


¢ Participants 
LJ} 29 male students aged between 18-30 
LJ All received a 4 hour training session 
LJ Score of 280% on a simulation-related competency test 


Results: Factors occur together 


Significance of p<0.001 


Negatively related, significance level 
of p<0.001 


NN — _ Significance of p<0.005 


Stress (SACL) 


0 2 3 4 5 
Workload (ISA) 


Mean STCAS 


Results: Co-occurring factors-a cumulative effect? 


Factors may combine in a cumulative way and 
associate with poorer performance 


2.5 


High WL, Good SA Low WL, Good SA_ High WL, Poor SA Low WL, Poor SA 
n=/ N=8 Factor Dyads "=9 n= 


Results (2) Time on task: Less resilient performance 


Low WL, Low Low WL, High WL, High WL, 
Stress n=10 High Stress Low Stress _ High Stress 
n=3 n=5 n=11 


High WL, LowWL, LowWL, ~~ High WL, 
Low Stress Low Stress High Stress High Stress 
n=5 n=10 n=5 n=9 


Behavioural markers of performance limits 


¢ Apparent link between some behaviours and 
self reported measures 
—Example: Indicators associated with fatigue 


— Yawning 
— Looking away from screen 
— Posture changes 


¢ Interviews 
— 22 ATCOs took part (17 males, 5 female) 
— What markers have you used that informed you 


about your performance ? 
Edwards et al., 2014 


Key findings 


¢ Controllers use internal and external markers 


“If someone’s getting stressed they can get louder or sit closer to 
the screen or something so if you see these things then you 
pay more attention yourself.” 


Edwards et al., 2014 


Markers are used to indicate edges of performance 


External -overt, observable 


Internal — subjective to the 
controller 


iidicators 


Lompens att Phrrsical Petformarice 
on strates charge decline 


Key findings 


¢ Controllers use internal and external markers 


¢ Markers are similar between controllers 


Edwards et al., 2014 


Key findings 


¢ Controllers use internal and external markers 
¢ Markers are similar between controllers 


¢ Developed from experience 


“You start to know that you’ve been burning your fingers before 
on this kind of situation and you really have to pay attention” 


Edwards et al., 2014 


Key findings 


Controllers use internal and external markers 
Markers are similar between controllers 


Developed from experience 


Markers are used to support performance 


“I'd say 300%, if you know that you're not being top performing today 
then that’s fine, just adapt your working style. But if you don’t know 
it, it might end in tears” 


Edwards et al., 2014 
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Key findings (2) Markers are used to support performance 
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Edwards et al., 2014 


Conclusions at the end of this research... 


¢ Multiple factor relationships: 


— Multiple factors co-occur to influence controller 
performance 


— Interactions between factors may create a cumulative 
influence on performance 


— But limitations of study challenge generalisability of 
results 


¢ Behavioural markers: 
— Markers indicate limits of performance 
— Controllers use markers to support performance 


Edwards et al., 2014 


Research overview 


* Overall Aims 


— ldentify the effect of automation in the ATC task on: 
¢ Workload 
e SA 
¢ Performance 


— Identify and verify interactions that threaten performance 


e Potential Outcomes 


— Better understanding of ‘difficult’ human performance factors in Air 
Traffic Control (ATC) 


— Signs and symptoms that performance is threatened 


Method: Simulation 


¢ Human in the loop, en-route 
high fidelity simulation (Part 


task) 
— Single high-altitude sector — 
in Cleveland ARTCC (79) _ ™* as Ze 
— Mix of level flight and | 
transitioning aircraft 
— No winds 


ZOB66 125.4 FL3304 
ZOB14 126.7 FL3204 


— All aircraft CPDLC equipped 


— All aircraft FMS and ADS-B 
equipped 


WP INIVERSITY 


Method: Design (1) 


¢ Within subjects design 
¢ Conducted as part of a larger study 


¢ 4task sets, Decreasing levels of automation: 
— Task set 1: Conflict detection only (CD) 
— Task set 2: Conflict detection and routine tasks (CD+RT) 


— Task set 3: Conflict detection, coordination and pilot 
requests — decision making (CD+DM) 


— Task set 4: Conflict detection, routine tasks, coordination's 
and pilot requests (CD+RT+DM) 


¢ Conflict probe running, but hidden 


meeee”CNNIVERSITY 


Method: Design (2) 


e Measures: 


Measure ISA SPAM Time to correctly detect 
conflicts 


Interval 3 3 Continuous 
(Mins) 


¢ Participants 
— 8 retired controllers from ZOA staffing the test sector 
— Age range 50-69 
— Experience in en-route ATC ranged from 23 — 29.5 years 
(M=24.94 SD=2.54) 
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Results 


¢ Workload significantly different between 
conditions 


— Task 1 — lowest workload 
— Tasks 2 and 4* highest 
¢ SA response times significantly different between 
conditions 
— Times slowest task 1 and task 3 
— Fastest task 2* and task 4 
¢ Time to detect conflicts significantly different 
between conditions 
— Slowest in task 1, fastest in task 2 
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Results (1): Automation significantly affects controller 
workload 
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(1) Conflict (2)CD+Routine (3)CD+Decision (4)CD+RT+DM 
Detection (CD) Tasks (RT) Making (DM) 


Condition 
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Results (2) Automation significantly attects 
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Results (3) Automation significantly affects controller 
performance 
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Results (3) Automation significantly affects controller 
performance 
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Results —Factor interactions: Task set 1 
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Results(2) —Factor interactions: Task set 2 
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Conclusions 


¢ Factors that influence controller performance 
(e.g. workload, fatigue) co-vary and appear to 
interact to create cumulative effect on 
performance 


¢ Results appear to be confirmed in a second 
experiment with a small, but expert, sample 

¢ Factor influences on performance may change 
with control context — e.g. automation 


Future directions 


HF Expert workshop 

— AHFE 2016 

— Concept development and (face) validation 
Collaboration between Future Sky and NASA Ames 


— Parallel development of human performance envelope model 
for pilots and controllers 


— Collaboration of Europe and US research 

Controlled simulations with expert participants 

— Part task and high-fidelity 

Factor scaling 

Further specification of edges of performance envelope 
— Markers 

— Psychophysiological measures? 


Implications 


Findings support a shift towards research investigating 
multi-factor co-occurrences and performance associations 


Training in markers 


— Predictive measures of human performance and prevention of 
performance decline 


Multifactor relationships - Performance prediction 
— Mitigation in the control room 
— Prevention of multifactor combinations 


Design of controller workstation/flight deck 


Adaptive automation implications 


Thank you! 


Back up slides 


Back up slides 
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Conflict Detection Study 


¢ How well can controllers detect conflicts ? 
— ,.when it’s their only responsibility ? 


— Could the addition of a secondary task impact their 
performance? 
¢ Routine tasks, such as hand-offs, check-ins, and frequency 
changes 
¢ Decision-making tasks, such as responding to flight crew 
requests or coordination requests from other controllers 
— 4x2x2 within-subjects experiment design 


¢ Primary independent variables: 
— Task set 
— Traffic density 
— Run length 


Conflict Detection Study 


+ 4x2x2 matrix [TASES TT Teer ener 
sual 


1x 1.2x | 20m 60m 


1x 1.2x | 20m 60m 


1x 1.2x | 20m 60m 


1x 1.2x | 20m 60m 


Conflict Detection Study 


¢ Simulation backdrop: 


— Single high-altitude sector 
in Cleveland ARTCC : 
— Mix of level flight and | m\ ZO 
transitioning aircraft 
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— Constant winds at altitude 
with forecast error 


GHOST a, 
ZOB66 125.4 FL3304 
ZOB14 126.7 FL3204 


— Conflict probe running, 
but hidden 


Conflict Detection Study 


¢ Dependent measures: 
— Controller detections are compared to the conflict 
probe data, producing: 
¢ Correctly identified conflicts 
¢ False alerts 
¢ Missed alerts 
— Real-time subjective workload ratings 
— Safety (separation violations) 


— Feedback from questionnaires and debrief 


Conflict Detection Study 


¢ Participants: 
— 8 retired controllers from ZOA staffing the test sector 


— 4 retired controllers from ZOA staffing the 
confederate airspace 

— 12 aviation students / general aviation pilots staffing 
the pseudo pilot positions 
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indicate edges of performance 


External -overt, observable 


Internal — subjective to the 
controller 


iidicators 


Lompens att Phrrsical Petformarice 
on strates charge decline 


Key results 


¢ Factors correlated as expected 


¢ Factor interactions associated with a 
Significantly larger performance decline 
compared to single factors 


¢ Significant relationships between observed 
participant behaviours and self-report 
measures 


ATC Exercise: Overvie ‘ 


Aims: 

LJ Investigate multifactor relationships and association® 
performance 

LJ Identify markers of performance edge 


Experiment: designed to reflect ATC working session 
_} 116 minute task (20 minute break after 60 minutes) 
_}] Task used real sectors, routes and traffic flow data 


UL] Taskload varied every 20 minutes between low and high ™™ 
through number of aircraft and complexity 


Measures: arousal, fatigue, SA, stress, workload 
Participant behaviours observed and recorded 


Behavioural markers of performance 
limits 
¢ Apparent link between some behaviours and 


self reported measures 
—Example: Indicators associated with fatigue 


— Yawning 
— Looking away from screen 
— Posture changes 


¢ Interviews 
— 22 ATCOs took part (17 males, 5 female) 
— What markers have you used that informed you 


about your performance ? 
Edwards et al., 2014 


Key findings 


¢ Controllers use internal and external markers 


“If someone’s getting stressed they can get louder or sit closer to 
the screen or something so if you see these things then you 
pay more attention yourself.” 


Edwards et al., 2014 


Key findings 


¢ Controllers use internal and external markers 


¢ Markers are similar between controllers 


Edwards et al., 2014 


Key findings 


¢ Controllers use internal and external markers 
¢ Markers are similar between controllers 


¢ Developed from experience 


“You start to know that you’ve been burning your fingers before 
on this kind of situation and you really have to pay attention” 


Edwards et al., 2014 


Key findings 


Controllers use internal and external markers 
Markers are similar between controllers 


Developed from experience 


Markers are used to support performance 


“I'd say 300%, if you know that you're not being top performing today 
then that’s fine, just adapt your working style. But if you don’t know 
it, it might end in tears” 


Edwards et al., 2014 


Markers of workload 


¢ Different markers for high workload and low 
workload: 


e Henwerktoac: 
Cat&gory Internal Marker Category External Marker 


Don't know the next steps Perception changes Can’t talk to executive/ 


Cognitive executive doesn’t hear you 
isib] Fi 
EMIS Calls are a surprise iced 


Redood setawarenews | |__| Movelosertoseren 
Subjective feeling | Colleagues not talking 


Markers of workload 


¢ Low workload: 


Category Internal Marker Proposed category | External Marker 
Perception changes Incorrect assessment of a 
situation 
Cognitive changes | Easily distracted 
Visible cues Sit back in chair 
Reduced self-awareness Away from radar screen 


Leave situations develop Talking to colleague 


. Performance changes Overlooking aircraft 
Trying to create more 


Changes to control 


complex situations Forgetting aircraft 


aii kan Aseria Repeated ‘sloppy’ mistakes 
a ; Fall behind traffic due to 
Subjective feeling 


Markers of fatigue 


Markers internal to the controller Observable markers 
experience 


Concentration issues More effort to control 
Laid back Not as confident 


Increased Don't want to work i a i 
assumptions busy trattic Falling asleep Distracted 


Slower force selfto pay | | Style of control 
attention 


nant : Less flexible Overlook aircraft 
Mi | Feel ti ae 
Mtulple, small 
Reduced awareness Not looking forward efficienc mistakes 
to shift ‘Running behind 


Less safety buffer traffic 


Incorrect plan 
problems 


Slower 
— Forget aircraft 
communications 


Markers of stress 


¢ Differentiation between positive stress and 
negative stress 


“It’s almost excited because there is more traffic coming. It’s a different 
situation if someone is already in a complex situation, you realise he ts 
falling behind 


Category Internal Marker 
Cognitive changes | Start to think slower 


Physiological Heartbeat 


Category External Marker 


Fidgeting 


Visible cues Red cheeks/neck 


changes Sweat Flustered 


Not coping Changes to voice Speaks faster, louder 


Feeling uncomfortable Speaks higher 


Anxious (negative) Demeanour Easily frustrated 


Subjective feeling 


Nervous Angry/confrontational 


Tense Blame others 


Markers of vigilance 


Category Internal Marker Category External Marker 
Performance changes Overlook aircraft 


Don't hear/see 
Surprised 


Cognitive/ Assume more 


erception changes = 
P P e Focused, ‘tunnel vision’ 
Donut effect 


Scan differently 
Changes to control 
Not leaving a problem 


Markers of losing the picture 


¢ Differentiation between markers that indicate 


losing the picture, and having lost the picture: 


“It starts off by just falling behind a bit. So you might just be a few steps 
behind what you’re supposed to be doing and if that builds up too much 
then you will get to the point where you start to lose the picture” 


Category Internal Marker Category External Marker 


priorities 


Performance changes Running behind 
iti Thinking whilst givin 
i a pce Time working ahead degrades 
the clearance = 
Missing calls 
Tunnel vision/hearing 


Subjective feeling Under confident 


Markers of having lost the picture 


Category Internal Marker Category External Marker 
Visible cues Zig zagging head movement 
Lose awareness 


of where to look 
Everything a surprise ‘Blacked out’/ silent 
Cognitive changes . 
Asking for confirmation 


Performance changes Unsafe clearance 
Cannot see a solution 


Unexpected decisions 
hanges to control | Reactive control Jumping from one aircraft to 
ee another 
Don't know who’s calling 


Don't react correctly 


Inadequate communications 


¢ Inadequate communications were described 
in relation to causes and contributory factors 
such as fatigue, lack of attention, or stress: 


“Mixing call signs happens more if someone’s tired or under pressure” 


“If you have aircraft that aren't listening and you’re busy...maybe the extra 
thing that sends you over” 


Category External Marker 


Situational issues Inadequate communications 
with aircraft 


Equipment failures 
Performance changes Mixing call signs 
Slip of the tongue 


Conclusions 


¢ Multiple factor relationships: 


— Multiple factors co-occur to influence controller 
performance 


— Interactions between factors may create a 
cumulative influence on performance 


¢ Behavioural markers: 
— Markers indicate limits of performance 
— Controllers use markers to support performance 


Edwards et al., 2014 


